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METHOD FOR PROCESSING IMAGE DATA T^D MODELING DEVICE 

[oool] This application is based on Japanese Patent 
Application No. 2001-033179 filed on February 9, 2001, the 
contents of which are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

1 , Field of the Invention 

[0002] The present invention relates to a method and 
device for processing an image in which a portion 
corresponding to a subject is extracted from plural shot 
image . 

2. Description of the Prior Art 

[0003] In various fields such as a computer graphics , an 
industrial design, or making of 3-D web contents, a three- 
dimensional data input system is used for digitizing a 
solid shape of an object. 

[0004] One of three-dimensional data input methods is 
called a shape -from- silhouette method. In this method, 
plural images obtained by shooting an object are used for 
extracting a shield contour (i.e., a silhouette) form each 
of the images, and a three-dimensional shape of the object 
is estimated from the extraction result and information of 
shooting positions. As an example of the application of 
the shape-from- silhouette method, Japanese patent 
publication No. 61-138377 discloses a technique for 
reconstructing a shape of an object by disposing a virtual 
object made of plural shape elements called voxels 
gathering in a three-dimensional virtual space, 
calculating a common portion of visual volumes determined 
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by eye points of shooting and the shield contours, and 
connecting the voxels corresponding to the common portion. 

[0005] Accuracy In reconstructing a shape of an object 
using the shape-from- silhouette method depends on how 
accurately the shield contours are extracted from each 
image. Under present circumstances, it is difficult to 
obtain a good result of extraction that can satisfy the 
demand of a user by completely automatic image processing, 
and an instruction or a correction is required to the user. 
Using a sophisticated image processing technique that 
utilizes knowledge of an object and a background, the 
accuracy of the extraction will be improved to some extent. 
However, it will require much time for inputting the 
knowledge, the object is limited, or the system is 
expensive. Moreover, even if a technique of extracting 
shield contours from shot images by disposing a screen of 
E a uniform color as a background for color clustering, a 

partial error of extraction cannot be avoided since real 
background color is not uniform due to a shadow or 
unevenness of lighting. 

[00O6] On the other hand, in order to reproduce a shape 
of an object precisely and correctly by the shape-from- 
silhouette method, as many portions as possible of a 
surface of the object should appear as shield contours in 
the images. A surface having a large curvature should be 
shot from various positions. Therefore, the number of 
necessary images becomes extremely large for an object 
having a complicated relief. 

[0007] Conventionally, there is a problem that a 
processing time necessary for extracting the shield 
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contours increase proportionally to the number of images 
necessary for reconstructing the shape of the object, and 
that a burden of the user who corrects the result of the 
extraction increases. 

SUMMARY OF THE INVENTION 
[ooosl An object of the present invention is to shorten 
a total time necessary for extracting an area for plural 
images . 

[0009] According to one aspect of the present invention, 
a method for processing image data is provided. The 
method comprises the steps of performing area extraction 
for each of the plural image sequentially. When 
performing the area extraction of the second and 
subsequent images, the result of the previous area 
extraction is reflected so as to determine the area to be 
extracted. As the order of the extraction process becomes 
later, the extraction results of more images are reflected, 
so that the area of the extraction target is narrowed and 
the extraction process time becomes short. Consequently, 
the total time necessary for area extraction of plural 
images is shortened compared with the case where the area 
extraction is performed individually without considering 
relationship between each image and other images. 

CoOlO] The reflection of the extraction result is 
realized by projecting a visual volume determined by the 
extracted area and the eye point when the image was 
obtained onto the image to be processed. Namely, the 
method comprises the steps of arranging the plural images 
in a three-dimensional virtual space in synchronization 



with the positional relationship between the object to be 
shot and the shooting position corresponding to each of 
the plural images, and checking which part of the other 
image the visual volume obtained by the area extraction 
for a certain image corresponds to. The visual volume is 
a cone-like space enclosed by line of sights passing 
through a shield contour, and the object exists within 
this visual volume. Therefore, there is always an area to 
be extracted within the area of the projected image of the 
visual volume. Thus, the target area of the area 
extraction is limited to the part of the image on which 
the visual volvmie is projected. In order to improve the 
accuracy of the projection, it is necessary to obtain an 
image formation condition (an optical parameter) for 
shooting along with information of the shooting position. 
If all the images are obtained by one camera, the optical 
parameter of the camera is necessary. If plural cameras 
are used, the optical parameters of the cameras are 
necessary. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[ooill Fig. 1 is a diagram showing a modeling system 

according to the present invention. 
[0012] Fig. 2 shows a screen structure of a monitor 

display . 

[0013] Fig. 3 is a flowchart showing a general procedure 
of modeling. 

[0014] Figs. 4A and 4B show a concept of a method for 
extracting an object area. 
[0015] Fig. 5 shows a concept of update of a three- 
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dimensional shape model. 

[ooie] Fig. 6 is a flowchart showing a detail procedure 
for extracting the object area. 

[0017] Fig. 7 is a flowchart showing a detail procedure 
of projection of a visual volume. 

[OOI8] Fig. 8 shows a functional configuration of an 
image data processor and a modeling device. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Q [0019] Hereinafter, the present invention will be 

2 explained more in detail with reference to embodiments and 

Cli drawings . 

m 

p [0020] Fig. 1 is a diagram showing a modeling system 

^ according to the present invention. 

O [0021] The modeling system 100 comprises a digital 

O camera 10, a lighting lamp 20, a rotating table 30, a 

J," camera calibration tool 40, a rotation axis calibration 

rU tool 50 and a computer 60. The digital camera 10 is used 

for taking picture of an object 90 placed on the rotating 
table 30. The shooting is performed plural times while 
changing positional relationship to the object 90. Plural 
images obtained by shooting the object 90 are transmitted 
to the computer 60, which uses them for modeling the 
object 90. Pixels of the image indicate color vector 
values of red, green and blue colors. The computer 60 is 
a data processing system including a body 61, a display 62, 
a keyboard 63 and a mouse 64. A program recorded in a CD- 
ROM 67 or another medium 68 is installed into the computer 
60. The program makes the computer 60 work as an image 
data processor and a modeling device. The image data 



processor includes three blocks 601, 602 and 603 as shown 
in Fig. 8 and extracts an object area corresponding to the 
object 90 from each of the plural images. The modeling 
device includes blocks 601, 602 and 603 and a block 604 
and reconstructs the shape in accordance with the 
extracted object areas. The block 601 acquires shoot 
condition data Dl that indicate the positional 
relationship between the digital camera 10 and the object 
90 when shooting and an optical parameter for image 
formation. The block 602 estimates the object area in one 
image in accordance with the object area extracted from 
another image and the shoot condition data Dl. The block 
603 corrects the object area in accordance with 
operational input. The block 604 estimates the three- 
dimensional shape of the object 90 in accordance with the 
object areas extracted from plural images and corrected 
and the shoot condition data Dl. Between the digital 
camera 10 and the computer 60, data can be transmitted 
online using a cable or an infrared communication or 
offline using a portable recording medium. 

[0022] The rotating table 30 is painted with a known 
color (Cr). The rotating table 30 is disposed in front of 
the tool 40, and the rotation angle thereof is controlled 
by the computer 60 via a cable. The user of the modeling 
system 100 can adjust the rotation angle of the rotating 
table 30 by operating the mouse 64. The tool 40 is a 
solid object whose surface is painted with a known color 
(Cb) on which thin lines are drawn in a grid using another 
color. The relationship between three-dimensional 
positions of each of the grid points of the tool 40 and a 



reference position of the tool 40 is known. The three- 
dimensional coordinate system whose origin is the 
reference position of the tool 40 defines world 
coordinates in the virtual space established on the memory 
of the computer 60 . The digital camera 10 is supported by 
a tripod and is placed so that the object 90 or the tool 
50 on the rotating table 30 is within the angle of view. 
In the online data transmission, the computer 60 receives 
a color image from the digital camera 10 responding to the 
mouse operation of the user. The position and posture of 
the digital camera 10 can be changed by moving the tripod, 
and the angle of view in shooting can be set by zoom 
adjustment. The tool 50, similarly to the tool 40, is a 
solid object whose surface is painted with a color on 
which thin lines are drawn in a grid using another color. 
Also in tool 50, the relationship between three- 
dimensional positions of each of the grid points and the 
reference position is known. The three-dimensional 
coordinate system whose origin is the reference position 
of the tool 50 defines object coordinates with respect to 
the object 90. Since the rotation of the rotating table 
30 is equivalent to the relative movement of the camera to 
the object 90, the relative movement between the camera 
coordinates on the basis of the eye position of each 
shooting can be described using the object coordinates. 
The lighting lamp 20 is disposed for lighting the rotating 
table 30 so that no shade is formed on the object 90 or 
the tool 50 . 

[0023] Fig. 2 shows a screen structure of a monitor 
display. The display screen 620 of the display 62 has two 



display areas (windows) 621 and 622. The display area 621 
displays an image 70 inputted from the digital camera 10 
or a composed image of the image 70 on which an extracted 
area is overlaid that is a result of an area extraction 
process that will be explained later. The extracted area 
can be corrected if necessary when the user erases or adds 
a unit area using the mouse 64. In addition, the display 
area 622 displays a three-dimensional shape model 80 with 
texture mapping on the process or after the process of 
reconstruction in the virtual space. The user can modify 
the position, the posture, the angle of view or others of 
the virtual camera in the virtual space that views the 
three-dimensional shape model 80 by operating the mouse. 
In addition, the user can select a shading display instead 
of the texture mapping display. In the shading display, 
the position of the virtual light source in the virtual 
space can be set freely by operating the mouse. 

[0024] Fig. 3 is a flowchart showing a general procedure 
of the modeling. 

[0025] [1] In Step #100, setting of the tool 50 is 
performed. The user places the tool 50 on the rotating 
table 30 so that at least two faces of the tool 50 are 
within the field of view of the digital camera 10. 

[0026l [2] In Step #102, the tool 50 is shot. The 
rotating table 30 is set to the state having the rotation 
angle Go for shooting, and then the rotating table 30 is 
rotated while fixing the eye position of the digital 
camera 10 without moving the same for shooting in the 
state having the rotation angle 60'. Thus, two images lo 
and lo' are obtained. 
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[0027] [3] In Step #104, correction of the rotation axis 
is performed. In the state where the image lo is 
displayed, the user operates the mouse 64 so as to put the 
cursor on one of the grid points of the camera calibration 
tool 40, clicks the mouse and inputs a predetermined grid 
point number. After repeating this operation for plural 
grid points, the user instructs the computer 60 to 
calculate a projection matrix Po that gives a geometric 
1^ relationship between the world coordinates and the pixel 

coordinate of the image lo. An example of the calculation 
method is described in the third chapter of "Three - 
Dimensional Computer Vision-A Geometric Viewpoint" by O. 
Faugeras, MIT Press, 1996. The computer 60 calculates 3 x 
4 projection matrix Po from the world coordinates at the 
rotation angle 60 and decomposes the projection matrix Po 
into a 3 X 3 internal parameter matrix Aq, a 3 x 3 
rotation matrix Ro and a three-dimensional parallel 
movement vector to. Namely, this process is equivalent to 
calculate the projection matrix Po, the internal parameter 
matrix Ao, the rotation matrix Rq and the parallel 
movement vector to satisfying the following equation when 
(Xw ,Yw ,Zw) denotes the value of the world coordinates and 
(xo, Yo) denotes the value of the image coordinates of the 
camera having the rotation angle 9o. 









Xq 


= Po 




1 




1 



(1) 

Po-Ao[Ro to] 

[0028] Thus, the conversion of the world coordinates (X^, 
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Yy,, Z„) into the camera coordinates (Xo, Yo, Zq) by the 
camera having the rotation angle 60 is given as follows, 
so that Ro and to are stored in the memory of the computer 



Xo 








Yo 








ZoJ 









rji 
ill 
01 

o 

m 



[0029] Next, similarly to the case of Pq, for the image 
lo, the projection matrix Oq from the object coordinates 
at the rotation angle 60, the internal parameter matrix Bo, 
the rotation matrix Qo, and the parallel movement vector 
So are calculated. 

[0030] In the same way, for the image lo', the 
projection matrix Oo' from the object coordinates at the 
rotation angle 80 ' . the internal parameter matrix Bo ' , the 
rotation matrix Qo ' , and the parallel movement vector So ' 
are calculated. The camera coordinate expression of the 
point (Ox, Oy, Oz) on the rotation axis by the camera 
having the rotation angle 60 is calculated as (o^, Oy, Oz) 
that satisfies the following equation. 



Ox 
Oy 

1 

T = 



Go' Go -Qo'QWso' 
o 1 



(3) 



[003l] Here, the fourth vector (Ox, Oy, Oz, 1)* 
corresponds to an eigenvector that corresponds to an 
eigenvalue "1" of the matrix T. Furthermore, the camera 



coordinate expression of the unit normal vector (nx, ny, 
nz)* of the rotation axis by the camera having the 
rotation angle 60 is calculated as a unit eigenvector 
corresponding to the eigenvalue 1 of the matrix Qo'Qo** 
when the sign of the calculated vector (n^, ny, n^)* is 
reversed if the sign of the value calculated with the 
following equation is different from sin (So '-So ). 
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[0032] The calculated parameters (n^, ny, nz) and {Ox, Oy, 
- Oz) of the rotation axis are stored in the memory of the 

o 

ny computer 60 . 

[0033] [4] In Step #106, setting of the object is 
O performed. The object 90 is placed on the rotating table 

m 

30 so that the entire object 90 is within the field of 
view of the digital camera 10. The object 90 is fixed 
using a tape or others if necessary, so that the object 90 
remains in a stationary position. 

[0034] [5] In Step #108, the three-dimensional shape 
data are initialized. In the virtual space, voxels whose 
sizes and quantities are designated by the user are 
arranged in a three-dimensional manner. On this occasion, 
the area where the voxels are arranged is set sufficiently 
larger than the area of the virtual space in which the 
object 90 is expected to exist. 

[0035] [6] In Step #110, the camera position is set. 
The user determines the positional relationship between 



the object 90 and the digital camera 10 by changing the 
position or the posture of the digital camera 10 or by 
rotating the rotating table 30. On this occasion, the 
computer 60 automatically stores the rotation angle 9i of 
the rotating table 30 in the memory. 

[0036] [7] In Step #112, shooting is performed. Using 
the digital camera 10, the user takes a picture of the 
object 90 so as to obtain the i-th (1=2, 3, 4, ....) 
image Ii of the object 90. 

[0037] [8] In Step #114, correction of the camera is 
performed. Similarly to the process in Step #104, 
concerning the image I^, the user operates the mouse 64 
for designating plural grid points of the tool 40 for 
correcting the camera, and the computer 60 executes the 
following calculation. First, a 3 x 4 projection matrix 
Pi' from the world coordinates at the rotation angle 0o of 
the camera having a new eye point is calculated, and the 
projection matrix Pi' is decomposed into a 3 x 3 internal 
parameter matrix Ai, a 3 x 3 rotation matrix Ri' and a 
three-dimensional parallel movement vector tl' . Then, 
considering the rotation of the rotating table 30, a 
projection matrix Pi at the rotation angle 61 of the camera 
having an eye point i is calculated as follows, so that 
the projection matrix Pi and parameters Ai, Ri and ti 
obtained by decomposing the projection matrix Pi are 
stored in the memory of the computer 60. 




-13- 



•cos ^r6o)]ny \n. Uy wJ+sinfe,— ^q) 



0 n^ -Hy 
fly -n^ 0 



(5) 



Si-[ I J Oy 



C0038] [9] In Step #116, the object area is extracted. 
As shown in Fig. 4A, for the image 72 noted as an 
extraction process target (the i-th image the visual 

voliime V71 enclosed by the lines of sight passing the 
object area 710 in the processed image 71 [j-th (j = 1, 
2. ..... i-1) image Ij] and the boundary thereof from 

outside (the shield contour) is projected, and the 
projection image area 721 in the image 72 is calculated. 
Color discrimination of the projection image area 721 is 
performed for extracting the object area 720, and the 
extraction result is displayed on the above-mentioned 
display area 621 of the display 62. The user corrects the 
extraction result by operating the mouse, if necessary. 
In this way, if the object area is already extracted from 
the plural images when performing the extraction process 
about the noted image, the plural images are referred. 
Namely, the plural visual volumes are projected onto the 
noted image, so that the area where the projection image 
areas overlap one another is determined. Then, color 
discrimination is performed for the area where the 
projection image areas overlap each other so that the 
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object area is extracted. In Fig. 4B, the visual volume 
V73 corresponding to the object area 731 in the image 73 
and the visual volume V74 corresponding to the object area 
741 in the image 74 are projected onto the image 75. The 
object area 750 in the image 75 is completely included in 
the overlap area 757 where the projection image area 753 
corresponding to the visual volume V73 and the projection 
image area 754 corresponding to the visual volume V74. 

[0039] [10] In Step #118. the three-dimensional shape 
data are updated. Using the projection matrix Pi 
corresponding to the image Ii that was required in the 
above-mentioned Step #114, all the voxels in the virtual 
space are projected onto the image Ii as shown in Fig. 5. 
Only the voxels included within the object area 720 
completely are remained, while all the other voxels are 
erased. 

[0040] [11] In Step #120, the three-dimensional shape 
data are displayed. A technique such as Marching Cube 
technique described in US patent No. 4710876, "System and 
method for the display of surface structures contained 
within the interior region of a solid body" is used for 
converting voxel data into polygon mesh. Corresponding to 
designation of the display form, the three-dimensional 
shape model is displayed with texture mapping or shading 
process . 

[004l] [12] In Step #122, it is decided whether the 
subsequent process should be performed or not. The user 
compares the three-dimensional shape model displayed on 
the display 62 with the real object 90 and instructs the 
computer 60 to stop the process if the shape is 
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sufficiently reproduced. If the user decided that the 
shape Is not sufficiently reproduced, he or she instructs 
the computer 60 to continue the process. In this case, 
the process goes back to Step #110 for taking a picture of 
another position or posture different from the previous 
one. 

[0042] Fig. 6 is a flowchart showing a detail procedure 
for extracting the object area. 

[0043l [1] In Step #200. the referred image is 
initialized. As a referred image that is used for 
extracting the object area, one of images whose object 
areas are already extracted is selected. 

[0044] [2] In Step #202, non-referred image is checked. 
It is checked whether all the images that can be referred 
to are used or not. If all the Images are already 
referred to, the process goes to Step #208. If there is 
an image that was not referred to, the process goes to 
Step #204. 

[0045] [3] In Step #204, the visual volume is projected. 
Tracing the boundary of the object area Sj of the image Ij 
from any point in the counterclockwise direction, lines of 
sight passing through the boundary are projected onto the 
image li one by one, and coefficient vectors m^i^ and m-ji' 
of the projection line of sight at both ends of the area 
where the line of sight projected onto the image 11 is 
scanned are stored in the memory of the computer 60. If 
plural object areas Sj are in the image Ij, the 
coefficient vectors mji'' and mji' of the projection line of 
sight at both ends are stored in the memory of the 
computer 60 for each of the object areas Sj^ (1 = 1, 
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2, ....,M-]) in accordance with the above-mentioned 
procedure. The detail will be explained later. 

[0046] [4] In Step #206, the next image to be referred 
is set. As the next image to be referred, one that can be 
referred to and is not yet referred to is selected. 

[0047] [5] In Step #208, overlap portion of projection 
image areas of the visual volume is determined. Among the 
pixels Pk (k = 1, 2, .....Nj) of the image Ii, a pixel area 
that satisfies either of the two following conditions is 
determined. This area becomes an intersection area v of 
the projection image areas, and the object area to be 
determined is included in this area. A binary image Hi 
having the pixel value "1" in the determined intersection 
area v and the pixel value "0" in the other area is made 
and is stored in the memory of the computer 60. 
Condition 1 : 

[x, y, ifm^O AND [;,^ ifm^O 
for all i=X2,..J-Xl = X2,...,Mj 

Condition 2: 

[xk y, ijwyi^o AND ]^Xk y, ljm>0 

for all j=X2,..J-%l=X2,...,Mj 

[0048] Here, it is supposed that the coordinate value of 
the pixel Pk "is given as (Xk, yk) • 

[0049] [6] In Step #210, color discrimination is 
performed. A pixel area where the pixel value 
corresponding to the binary image Hi is "1" and the pixel 
value corresponding to the image Ii satisfies the 
following condition is determined, and a binaary image Gi 
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having the pixel value corresponding to the determined 
pixel area of "1" and the other pixel value of "0" is 
generated, 

h-C,l>h OR ]p,-c\>tr (8) 

[0050] Here, reference characters Cb and Cr denote a 
color vector of the color Cb of the tool 40 and the color 
vector of the color Cr of the rotating table 30, 
respectively. The reference character Ck denotes the 
color vector of the pixel Pk- The reference characters tt 
and tr are threshold level constants given by the user. 

[005l] An opening process of a morphology operation is 
performed for the obtained binary image Gi, and the result 
of the process is saved. The pixel having the binary 
image value of "1" is a pixel that was estimated to belong 
to the object area. The opening process of the morphology 
operation is performed for eliminating grid lines drawn on 
the tool 40 from the extraction result. 

[0052] [7] In Step #212, the object area is displayed. 
An image Ij' is generated in which the pixel value of the 
area where the pixel value corresponding to the binary 
image Hi is "1" is the vector indicating the display color 
Co of the intersection area and the pixel value in the 
other area is the vector indicating the background display 
color Cg. Next, an image Ji is generated in which the 
color generated by mixing colors of pixels corresponding 
to the image I3' and the image Ij is the color of each 
pixel. Then, the boundary of the pixel area where the 
pixel value of the binary image Gi is "1" is drawn on the 
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image Ji with a dotted line, so as to display on the 
display area 621. 

[0053] [8] In Step #214, the user decides where 
correction is necessary or not. The user observes the 
display content of the display area 621. If he or she 
decides that the real object area is different from the 
extracted area whose boundary is indicated by the dotted 
line, he or she performs a predetermined mouse operation 
for instructing the computer 60 to go to Step #216. If 
the user decides there is no portion to be corrected, he 
or she instructs the computer 60 to go to the above- 
mentioned Step #118. 

[0054] [9] In Step #216, the object area is corrected. 
If the extracted area lies off the real object area, the 
user can set the pixel value of the binary image Gi 
corresponding to the area off the real object area to "0" 
by tracing the area with a cursor while dragging the mouse 
64. Moreover, if the user decides that the extracted area 
is partially short of the real object area, he or she can 
set the pixel value of the binary image Gi corresponding 
to the shortage area to "1" by tracing the area with a 
cursor while dragging the mouse 64. The image Ji can be 
changed in the same way as in Step #212. Usually, the 
intersection area v of the projected visual volume 
includes the real object area completely, so the user is 
not required to correct the area outside the intersection 
of the visual volumes, resulting in reduction of the 
user's burden. In addition, the area where the correction 
is not required becomes larger in general as later the 
area of the image is extracted. Therefore, as the area 
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extractlon is performed for each image, the user's burden 
of the correction is less . 

[0055] Fig. 7 is a flowchart showing a detail procedure 
of the projection of the visual volume. 

[0056l [1] In Step #300, the F matrix calculation is 
performed. The F matrix between the images Ii and Ij is 
calculated by the following equation. 



F =(A-;yT,R,Aj' 



0 -ty 

-t, 0 

ty -t, 0 



-Rjiti^tj 



(9) 



[0057] Here, F is a 3 x 3 matrix. A set of points (xi, 
Yi) on the image Ii that satisfies the following equation 
indicates a line obtained by projecting the line of sight 
passing through the image coordinates (Xj, yj) of the 
image 1 3 onto the image Ii. 



=0 



(10) 



[0058] This line e-j is called an epipolar line. The 
line of sight of the image Ii projected on the image Ii 
passes through a point on the image Ii called an epipole. 
The image coordinates of the epipole on the image Ii is 
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given as (ex/ez, ey/Sz), when (ex, ey, ez) denotes an 
eigenvector corresponding to the zero eigenvalue of the 
matrix F. However, when ez is zero, the eplpole is at the 
end point at infinity, so all the epipolar lines on the 
image Ii are lines parallel to the vector (e^, ey) . In 
addition, the epipole on the image Ij can be determined 
from the eigenvector corresponding to the zero eigenvalue 
of the matrix in the same way. 

[0059] [2] In Step #302, it is checked whether there is 
a non-processed boundary pixel or not. In the image I3, 
it is checked whether the boundary pixels of the object 
area are all processed or not. If all the boundary pixels 
are processed, the process goes back to Step #206. If 
there is a non-processed boundary pixel, the process goes 
to Step #304. 

[0O6O] [3] In Step #304, a non-processed boundary pixel 
is selected. In the image I3, any one of the boundary 
pixels is selected from the first non-processed object 
area Sj^. 

[0O6I] [4] In Step #306, the calculation of the epipolar 
surface is performed. The coordinates of the point in the 
three-dimensional space indicated by the camera 
coordinates of the image Ii is denoted by (Xi, Yi, Z±) . and 
the coordinate value of the boundary pixel selected in 
Step #304 is denoted by (xj, yj). Then, the equation of 
the plane determined by the line of sight that passes the 
point (xj, yj) of the image Ij and the camera eye point of 
the image Ii is calculated as follows. 



[xj yj i]f 



Zi\ 



=0 (11) 
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[0062] The plane Ej defined by this equation is called 
an epipolar surface. The intersection of the epipolar 
surface and the projection surface of the image is the 
epipolar line ej. The vector containing the elements that 
are coefficients of Xi, Yi, and Zi in this equation is made 
a unit vector n^, which is stored in the memory of the 
computer 60. The vector nj indicates the normal direction 
of the epipolar surface Ej. 

[0063l [5] In Step #308, the posture of the epipolar 

1=5= 

D surface is calculated. From the following equation, the 

O 

ip angle Gj between the normal of the epipolar surface Ej and 



ill 
P 



the optical axis of the camera of the image Ii is derived. 
COS^, = [0 0 l]n. (12) 

[0064] [6] In Step #310, the epipolar lines at both ends 
are updated. There can be an object area on the epipolar 
line ej on the image Ii corresponding to the epipolar 
surface E-j, so the both ends of the area where the 
epipolar line corresponding to the object area Sj-^ exists 
is traced as below. 

[0065] The epipolar line e^ corresponding to the 
epipolar surface Ej is given by the following equation 
concerning the parameter (xi, y±) • 



n)AT J, =0 
1 



(13) 



[0066] If the coefficient vectors mji* and mji" of the 
epipolar lines e^i* and eji" at both ends corresponding to 
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the object area Sj"^ are not saved, the coefficient vector 
of the epipolar line ej is saved as m-ji* and niji' 
unconditionally. Namely, the following equation is 
satisfied. 

m]rm'ji''^Af) (14) 



[0O67] At the same time, the angle 6j calculated in Step 
#308 is saved in the memory of the computer 60 as the 
angles Sn* and 9ii" of the epipolar surface corresponding 
p to the epipolar lines 63/ and Sji". Namely, the following 

m 

ffs equation is satisfied. 

5 

3 cosdji =cosdJi =cosdj (15) 

ji^ [0O68] If the coefficient vectors m-ji"^ and mji' of the 

-J; epipolar line at both ends are already saved, m^i^ and 811^ 

O 

ry are updated only when 6j is larger than Q±i , and mji" and 

611' are updated only when 63 is smaller than 611". Namely, 
the following equation is satisfied. 

m]i = injAl^) 'COS d]rcos dj cos d]i> cos 6j 
m'jrin'jAi^] ^cos 6}= cos dj if cos d'ji< cos dj ^^^^ 



[0O69] [7] In Step #312, a neighboring boundary pixel is 
selected. The next boundary pixel is selected by tracing 
the boundary in the counterclockwise direction from the 
selected boundary pixel. 

[00703 [8] In Step #314, it is checked whether the 
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boundary of the object area is traced one round or not. 
If the newly selected boundary pixel is already processed, 
all the boundary pixels of the first object area S-j^ are 
processed. Then, the process goes back to Step #302, and 
the process of the boundary pixels of other object areas 
is started. If it is not processed yet, the process goes 
back to Step #306. 

[007l] In the above-mentioned embodiment, there are 
several variations as follows . 

[0072] [Variation 1] It is possible that the colors of 
the tool 40 and the rotating table 30 are not known. Step 
#210 is omitted when extracting the object area, and the 
user extracts the object area by a predetermined operation. 
When adopting this variation, the user is required only to 
erase excess object area without adding a new object area 
since the intersection area of the projected visual volume 
involves the real object area completely. In addition, 
there is a merit of low frequency that the user is 
bothered by a complicated operational procedure. 

[0073] [Variation 2] It is possible that the colors of 
the tool 40 and the rotating table 30 are not known, and 
the color to be given as known is obtained by sampling in 
the image area outside the intersection area of the visual 
volume in Step #210. Thus, the system becomes resistant 
to the influence of variation of the lighting environment. 

[0074] [Variation 3] Instead of the configuration where 
the inner parameter and the position and posture parameter 
of the digital camera 10 is calibrated online using the 
tool 40 and the rotating table 30, it is possible to 
calibrate the inner parameter of the camera offline in 
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advance without the tool and to obtain the position and 
posture parameter by using a digital camera disposed at 
the tip of a robot arm as described in US patent No. 
05402582, "Three dimensional coordinate measuring 
apparatus". Alternatively, the digital camera 10 can be 
equipped with a position and posture sensor detecting a 
magnetic field as described in US patent No. 05307072, 
"Non - concentricity compensation in position and 
orientation measurement systems" or a combination of a 
gyroscope sensor and an acceleration sensor, so as to 
obtain the position and posture parameter. This variation 
can be combined with Variation 1, so as to process 
completely in any background. 

[0075] [Variation 4] In the above-mentioned example, 
onto the target image of the area extraction, the visual 
volume of the other image is projected, and then the 
intersection area of the projected image area is 
calculated. However, it is possible that the intersection 
of the visual volume in the three-dimensional space is 
determined, and then the three-dimensional volume is 
projected onto the image that is noted as a target of the 
area extraction. In other words, it is possible to 
exchange the execution orders of Step #116 and Step #118 
in Fig. 3, to project the three-dimensional shape data 
updated in Step #118 onto the noted image instead of Step 
#200 through Step #208, and to mark for indicating the 
pixel in the projection image area as a process target. 

[0076] [Variation 5] In Variation 4 the correction of 
the extracted object area is performed on a two- 
dimensional image. However, it is possible to correct the 
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object area indirectly by correcting the Intersection of 
the visual voltime itself in the three-dimensional space. 

[0077] [Variation 6] It is possible to use the result of 
the area extraction as information for tracking a position 
of the object in the image. In other words, when adopting 
the method (R. Cipolla and A. Blake, "The Dynamic Analysis 
of Apparent Contours," Proc. Third ICCV, 1990), in which 
sequential images are obtained by moving the camera 
continuously for shooting the object, the parameter in the 
previous image is used as an initial value for fitting the 
contour model called a snake into the contour of the 
object. In this case, the result of the area extraction 
is used as the initial value. 

[0078] [Variation 7] It is possible not to use the tool 
40 for calibrating the inner parameter and the position 
and posture parameter of the digital camera 10, and to 
perform the calibration of the parameters by determining 
any corresponding points of the object and the background 
between the images. 

[0079] While the presently preferred embodiments of the 
present invention have been shown and described, it will 
be understood that the present invention is not limited 
thereto, and that various changes and modifications may be 
made by those skilled in the art without departing from 
the scope of the invention as set forth in the appended 
claims . 



